Performance of the Ebel standard-setting method in spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisted of multiple-choice questions.

PURPOSE: It aimed to know the performance of the Ebel standard-setting method in in spring 2019 Royal College of Physicians and Surgeons of Canada internal medicine certification examination consisted of multiple-choice questions. Specifically followings were searched: the inter-rater agreement; the correlation between Ebel scores and item facility indices; raters' knowledge of correct answers' impact on the Ebel score; and affection of rater's specialty on theinter-rater agreement and Ebel scores. METHODS: Data were drawn from a Royal College of Physicians and Surgeons of Canada certification exam. Ebel's method was applied to 203 MCQs by 49 raters. Facility indices came from 194 candidates. We computed Fleiss' kappa and the Pearson correlation between Ebel scores and item facility indices. We investigated differences in the Ebel score (correct answers provided or not) and differences between internists and other specialists with t-tests. RESULTS: Kappa was below 0.15 for facility and relevance. The correlation between Ebel scores and facility indices was low when correct answers were provided and negligible when they were not. The Ebel score was the same, whether the correct answers were provided or not. Inter-rater agreement and Ebel scores was not differentbetween internists and other specialists. CONCLUSION: Inter-rater agreement and correlations between item Ebel scores and facility indices wee consistently low; furthermore, raters' knowledge of correct answer and rater specialty had no effect on Ebel scores in the present setting.

How Attending Physician Preceptors Negotiate Their Complex Work Environment: A Collective Ethnography.

PURPOSE: To generate an empiric, detailed, and updated view of the attending physician preceptor role and its interface with the complex work environment. METHOD: In 2013, the authors conducted a modified collective ethnography with observations of internal medicine medical teaching unit preceptors from two university hospitals in Canada. Eleven observers conducted 32 observations (99.5 hours) of 26 preceptors (30 observations [93.5 hours] of 24 preceptors were included in the analysis). An inductive thematic approach was used to analyze the data with further axial coding to identify connections between themes. Four individuals coded the main data set; differences were addressed through discussion to achieve consensus. RESULTS: Three elements or major themes of the preceptor role were identified: (1) competence or the execution of traditional physician competencies, (2) context or the extended medical teaching unit environment, and (3) conduct or the manner of acting or behaviors and attitudes in the role. Multiple connections between the elements emerged. The preceptor role appeared to depend on the execution of professional skills (competence) but also was vulnerable to contextual factors (context) independent of these skills, many of which were unpredictable. This vulnerability appeared to be tempered by preceptors' use of adaptive behaviors and attitudes (conduct), such as creativity, interpersonal skills, and wellness behaviors. CONCLUSIONS: Preceptors not only possess traditional competencies but also enlist additional behaviors and attitudes to deal with context-driven tensions and to negotiate their complex work environment. These skills could be incorporated into role training, orientation, and mentorship.

How evidence from observing attending physicians links to a competency-based framework.

CONTEXT: Competency-based medical education frameworks are often founded on a combination of existing research, educational principles and expert consensus. Our objective was to examine how components of the attending physician role, as determined by observing preceptors during their real-world work, link to the CanMEDS Physician Competency Framework. METHODS: This is a sub-study of a broader study exploring the role of the attending physician by observing these doctors during their working day. The parent study revealed three overarching elements of the role that emerged from 14 themes and 123 sub-themes: (i) Competence, defined as the execution of traditional physician competencies; (ii) Context, defined as the environment in which the role is carried out, and (iii) Conduct, defined as the manner of acting, or behaviours and attitudes in the role that helped to negotiate the complex environment. In this sub-study, each sub-theme, or 'role-related component', was mapped to the competencies described in the CanMEDS 2005 and 2015 frameworks. RESULTS: Many role-related components from the Competence element were represented in the 2015 CanMEDS framework. No role-related components from the Context element were represented. Some role-related components from the Conduct element were represented. These Conduct role-related components were better represented in the 2015 CanMEDS framework than in the 2005 framework. CONCLUSIONS: This study shows how the real-world work of attending physicians links to the CanMEDS framework and provides empirical data identifying disconnects between espoused and observed behaviours. There is a conceptual gap where the contextual influences of physicians' work and the competencies required to adjust to these influences are missing from the framework. These concepts should be incorporated into learning both broadly, such as through an emphasis on context within curriculum development for the workplace (e.g. entrustable professional activities), and explicitly, through the introduction of novel competencies (e.g. the Conduct role-related components described in this study).

Diagnosing technical competence in six bedside procedures: comparing checklists and a global rating scale in the assessment of resident performance.

PURPOSE: To compare procedure-specific checklists and a global rating scale in assessing technical competence. METHOD: Two trained raters used procedure-specific checklists and a global rating scale to independently evaluate 218 video-recorded performances of six bedside procedures of varying complexity for technical competence. The procedures were completed by 47 residents participating in a formative simulation-based objective structured clinical examination at the University of Calgary in 2011. Pass/fail (competent/not competent) decisions were based on an overall global assessment item on the global rating scale. Raters provided written comments on performances they deemed not competent. Checklist minimum passing levels were set using traditional standard-setting methods. RESULTS: For each procedure, the global rating scale demonstrated higher internal reliability and lower interrater reliability than the checklist. However, interrater reliability was almost perfect for decisions on competence using the overall global assessment (Kappa range: 0.84-1.00). Clinically significant procedural errors were most often cited as reasons for ratings of not competent. Using checklist scores to diagnose competence demonstrated acceptable discrimination: The area under the curve ranged from 0.84 (95% CI 0.72-0.97) to 0.93 (95% CI 0.82-1.00). Checklist minimum passing levels demonstrated high sensitivity but low specificity for diagnosing competence. CONCLUSIONS: Assessment using a global rating scale may be superior to assessment using a checklist for evaluation of technical competence. Traditional standard-setting methods may establish checklist cut scores with too-low specificity: High checklist scores did not rule out incompetence. The role of clinically significant errors in determining procedural competence should be further evaluated.

Safely and effectively reducing inpatient length of stay: a controlled study of the General Internal Medicine Care Transformation Initiative.

PURPOSE: Whether improving the efficiency of hospital care will worsen post-discharge outcomes is unclear. We designed this study to evaluate the General Internal Medicine (GIM) Care Transformation Initiative implemented at one of the seven teaching hospitals in the Canadian province of Alberta. METHODS: Controlled before-after study of GIM patients hospitalised at the University of Alberta Hospital (UAH, intervention site, n=1896) or the six other teaching hospitals in Alberta-three in Edmonton (intra-regional controls (IRC), n=4550) and three in Calgary (extra-regional controls (ERC), n=4095). The primary effectiveness outcome was risk-adjusted length of stay (LOS) and the primary safety outcome was 'mortality during index hospitalisation or all-cause readmission or death within 30-days of discharge'. RESULTS: LOS for GIM patients decreased by 0.68 days at Alberta teaching hospitals between 2009 and 2012; GIM patients hospitalised at the UAH exhibited a further 20% relative decline in adjusted LOS (total reduction=1.43 days, 95% CI 0.94 to 1.92 days) from PRE to POST. Interrupted time series (ITS) confirmed that the 1.43 day reduction at the UAH was statistically significant (level change p=0.003), while the declines at the IRC (p=0.37) and ERC (p=0.45) were not. Our safety outcome did not change for UAH patients (18.4% PRE-intervention vs 17.8% POST-intervention, adjusted OR 1.02 (95%CI 0.80 to 1.31), p=0.42 on ITS), nor for those hospitalised at the IRC (p=0.33) or the ERC (p=0.73) sites. CONCLUSIONS: The Care Transformation Initiative was associated with substantial reductions in LOS without increasing post-discharge events commonly quoted as proxies for quality.

Who is teaching and supervising our junior residents' central venous catheterizations?

BACKGROUND: The extent to which medical residents are involved in the teaching and supervision of medical procedures is unknown. This study aims to evaluate the teaching and supervision of junior residents in central venous catheterization (CVC) by resident-teachers. METHODS: All PGY-1 internal medicine residents at two Canadian academic institutions were invited to complete a survey on their CVC experience, teaching, and supervision prior to their enrolment in a simulator CVC training curriculum. RESULTS: Of the 69 eligible PGY-1 residents, 32 (46%) consenting participants were included in the study. There were no significant baseline differences between participants from the two institutions in terms of sex, number of ICU months completed, previous CVC training received, number of CVCs observed and performed. Only 16 participants (50%) received any CVC training at baseline. Of those who received any training, 63% were taught only by senior resident-teachers. A total of 81 CVCs were placed by 17 participants. Thirty-two CVCs (45%) were supervised by resident-teachers. CONCLUSIONS: Resident-teachers play a significant role both in the teaching and supervision of CVCs placed by junior residents. Educational efforts should focus on preparing residents for their role in teaching and supervision of procedures.

Development and validation of a cardiac findings checklist for use with simulator-based assessments of cardiac physical examination competence.

INTRODUCTION: Objective outcome measures for use with simulator-based assessments of cardiac physical examination competence are lacking. The current study describes the development and validation of an approach to scoring performance using a cardiac findings checklist. METHODS: A cardiac findings checklist was developed and implemented for use with a simulator-based assessment of cardiac physical examination competence at a Canadian national specialty examination in internal medicine. Candidate performance as measured using the checklist was compared with global ratings of clinical performance on the cardiac patient simulator and with overall examination performance. RESULTS: Interrater reliability for scoring the checklist ranged from 0.95 for scoring correct findings to 0.72 for scoring incorrect findings. A summary checklist score had a Pearson correlation of 0.60 with overall candidate performance on the simulator-based station. CONCLUSION: Use of a cardiac findings checklist provides one objective measure of cardiac physical examination competence that may be used with simulator-based assessments.

The effect of English language proficiency on length of stay and in-hospital mortality.

BACKGROUND: In ambulatory care settings, patients with limited English proficiency receive lower quality of care. Limited information is available describing outcomes for inpatients. OBJECTIVE: To investigate the effect of English proficiency on length of stay (LOS) and in-hospital mortality. DESIGN: Retrospective analysis of administrative data at 3 tertiary care teaching hospitals (University Health Network) in Toronto, Canada. PARTICIPANTS: Consecutive inpatient admissions from April 1993 to December 1999 were analyzed for LOS differences first by looking at 23 medical and surgical conditions (59,547 records) and then by a meta-analysis of 220 case mix groups (189,119 records). We performed a similar analysis for in-hospital mortality. MEASUREMENTS: LOS and odds of in-hospital death for limited English-proficient (LEP) patients relative to English-proficient (EP) patients. RESULTS: LEP patients stayed in hospital longer for 7 of 23 conditions (unstable coronary syndromes and chest pain, coronary artery bypass grafting, stroke, craniotomy procedures, diabetes mellitus, major intestinal and rectal procedures, and elective hip replacement), with LOS differences ranging from approximately 0.7 to 4.3 days. A meta-analysis using all admission data demonstrated that LEP patients stayed 6% (approximately 0.5 days) longer overall than EP patients (95% confidence interval, 0.04 to 0.07). LEP patients were not at increased risk of in-hospital death (relative odds, 1.0; 95% confidence interval, 0.9 to 1.1). CONCLUSIONS: Patients with limited English proficiency have longer hospital stays for some medical and surgical conditions. Limited English proficiency does not affect in-hospital mortality. The effect of communication barriers on outcomes of care in the inpatient setting requires further exploration, particularly for selected conditions in which length of stay is significantly prolonged.